Processing apparatus, image pickup apparatus, image pickup system, and processing method

ABSTRACT

A processing apparatus combines a plurality of images based on a plurality of object images formed on an imaging plane of an image sensor by a plurality of lens units and to generate a combined image, and includes at least one processor or circuit that serves as an acquisition task configured to acquire information on a center position of each of the plurality of object images on the imaging plane, information on a correspondence relationship between the center position and positions of the plurality of images in the combined image, and conversion information for converting a first coordinate system in the imaging plane into a second coordinate system in the combined image, the conversion information being generated based on a correction function for correcting the plurality of object images, and a processing task configured to generate the combined image using the conversion information.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/320,407, filed on May 14, 2021, which claims the benefit of andpriority to Japanese Patent Application No. 2020-097907, filed Jun. 4,2020, each of which is hereby incorporated by reference herein in theirentirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a processing apparatus, an image pickupapparatus, an image pickup system, and a processing method.

Description of the Related Art

A method of generating a combined image has conventionally been known byacquiring a plurality of images different from each other in singleimaging using an optical system that forms a plurality of images of thesame object with a plurality of lenses, and by staking (superimposing)the acquired plurality of images. Japanese Patent No. 5910739 disclosesan image pickup apparatus that includes a filter array and a lens arrayafter the objective lens, and can simultaneously acquire a plurality ofimages for generating a multiband image with a common image sensor.

In the image pickup apparatus disclosed in JP 5910739, a plurality oflens units in the lens array image light having different wavelengthsfrom each other, and cause the distortion in a plurality ofspectroscopic images and consequently a shift in positional informationamong the plurality of spectroscopic images. A correct result cannot beobtained by a spectroscopic analysis that uses the multiband imagegenerated by stacking the plurality of spectroscopic images in which thepositional information shifts.

SUMMARY OF THE INVENTION

The present invention provides a processing apparatus, an image pickupapparatus, an image pickup system, and a processing method, each ofwhich can generate a good combined image.

A processing apparatus according to one aspect of the present inventionis configured to combine a plurality of images based on a plurality ofobject images formed on an imaging plane of an image sensor by aplurality of lens units and to generate a combined image. The processingapparatus includes at least one processor or circuit configured toexecute a plurality of tasks including an acquisition task configured toacquire information on a center position of each of the plurality ofobject images on the imaging plane, information on a correspondencerelationship between the center position and positions of the pluralityof images in the combined image, and conversion information forconverting a first coordinate system in the imaging plane into a secondcoordinate system in the combined image, the conversion informationbeing generated based on a correction function for correcting theplurality of object images, and a processing task configured to generatethe combined image using the conversion information. At least oneprocessor or circuit is configured to perform a function of at least oneof the units. An image pickup apparatus and an image pickup systemincluding the above processing apparatus and a processing method for aprocessing apparatus corresponding to the above processing apparatusalso constitute another aspect of the present invention.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to 1D schematically illustrate an image pickup system accordingto a first embodiment.

FIGS. 2A and 2B explain a distortion correcting method according to thefirst embodiment.

FIGS. 3A to 3C schematically illustrate a data structure of a multibandimage.

FIG. 4 is a flowchart showing a method of converting an image acquiredin the first embodiment into a multiband image.

FIG. 5 schematically illustrates an image pickup system according to athird embodiment.

FIG. 6 explains an eccentric distortion.

DESCRIPTION OF THE EMBODIMENTS

Referring now to the accompanying drawings, a detailed description willbe given of embodiments according to the present invention.Corresponding elements in respective figures will be designated by thesame reference numerals, and a duplicate description thereof will beomitted.

First Embodiment

FIGS. 1A to 1D schematically illustrate an image pickup system accordingto this embodiment. FIG. 1A is a side view of the image pickup system,and FIG. 1B is a front view of the image pickup system viewed from theobject side. The image pickup system includes an optical system, animage sensor SS, and a processor 100. The optical system includes afilter array FA (omitted in FIG. 1B) and a lens array LA. The imagepickup system may include an optical apparatus including the opticalsystem, and an image pickup apparatus that includes the image sensor SSand the processor 100 and is mountable with the optical apparatus.

As illustrated in FIG. 1C, the processor 100 includes an acquisitionunit (acquisition task) 100 a and a processing unit (processing task)100 b. The processor 100 may further include a memory (not shown) forstoring information acquired by the acquisition unit 100 a andinformation generated (processed) by the processing unit 100 b. Althoughthe processor 100 is installed in the image pickup system in thisembodiment, it may be configured as a processing apparatus separate fromthe image pickup system. The processor 100 may be installed in the imagepickup apparatus or optical apparatus.

The lens array LA includes a 2×2 array lens unit (imaging unit) ML11,ML12, ML21, and ML22. This embodiment sets 2×2 lens units to the lensarray LA for simple explanation, but the present invention is notlimited to this example. Even when a plurality of lenses are disposed inan array, it serves as the lens array LA.

Since each lens unit is disposed and focused on the image sensor SS, thesame object image is imaged (formed) on the imaging plane of the imagesensor SS by the number of lens units. Since a plurality of objectimages are formed in a tile shape according to the arrangement of thelens units, the images formed by the lens units will be referred to as“tile images” in the following description. The tile image is developedaround an intersection as a center position between the optical axis ofthe corresponding lens unit and the image sensor SS.

The filter array FA is disposed on the optical axis of the correspondinglens unit, and includes a plurality of bandpass filters havingtransmission characteristics different from each other. Making thetransmission characteristics of the plurality of bandpass filtersdifferent from each other can develop a plurality of tile images of thesame object and light having different wavelengths on the imaging plane.In other words, the image pickup system according to this embodiment canacquire a plurality of spectral images at once (simultaneously). Sincethe number of tile images (number of bands) is proportional to thenumber of bandpass filters and lens units, the number of bandpassfilters and lens units may be increased or decreased according to therequired number of bands.

The processor 100 generates a multiband image (combined image) byvirtually stacking (superimposing or combining) a plurality of(spectroscopic) images based on a plurality of tile images acquired fromthe image sensor SS. A colored image acquired by a normal camera isgenerated by superimposing three-layer (R, G, B) spectroscopic images(two-dimensional luminance map) containing different color (spectralwavelength) information. On the other hand, the multiband image isgenerated by stacking spectroscopic images of more than three layers. Aninterval in the layer direction (wavelength direction) (increment of thespectral wavelength) corresponds to a wavelength resolution, and thenumber of layers (overall width of the wavelength) corresponds to awavelength range. Although it depends on the data structure, a regularinterval of the spectral wavelength and the spectral images arranged indescending or ascending order of the spectral wavelengths can provide aresult easier to use for the analysis. The spectroscopic images ofimportant spectral wavelengths may be arranged in front of the data toaccelerate the analysis, or an irregular interval of the spectralwavelength may be used.

Given information on a correspondence between information on the centerposition of the tile image (center position information) and spectralwavelength information (layer position of the spectroscopic image in themultiband image), the multiband image stacked in the wavelength orderfollows. This embodiment stores, as stack information, the informationon the correspondence between the center position information and thespectral wavelength information.

The configuration illustrated in FIG. 1A may need focusing of the lensarray LA depending on the object distance. FIG. 1D illustrates amodified example of the image pickup system. In FIG. 1C, a focusing lensL1 having a larger aperture is disposed on the object side of the lensarray LA. When an exit pupil of the lens L1 includes entrance pupils ofall lens units in the lens array LA, an imaging relationship of theoptical system is maintained. If this condition is satisfied, focusingcan be achieved by simply disposing a lens having a short focal lengthwhile the lens array LA is fixed relative to the image sensor SS,particularly in imaging a close object.

The distortion of the lens unit will now be described. When theplurality of lens units have the same design, each image has a chromaticaberration when the plurality of lens units image light that hastransmitted through different bandpass filters. In the chromaticaberrations, the lateral chromatic aberration is particularly observedas if the image height is expanded or contracted in comparison with eachtile image. In other words, the lateral chromatic aberration viewed inthe tile image is observed like the distortion, and image processing(image correction) cannot distinguish between the lateral chromaticaberration and the distortion. Therefore, the lateral chromaticaberration and the distortion in the (two-dimensional) spectroscopicimaging are collectively treated as the distortion. The multiband imagecan be generated by simply stacking a plurality of spectroscopic imagesacquired by the imaging sensor SS, but the distortion for each tileimage causes a shift in positional information among the spectroscopicimages. This shift appears as a color shift on the multiband image andmay deteriorate the image quality of the multiband image. Therefore,even when the multiband image is used for the spectroscopic analysis, acorrect result cannot be obtained.

A description will now be given of a method for correcting thedistortion. FIGS. 2A and 2B explain the distortion correcting methodaccording to this embodiment. FIG. 2A illustrates a distortioncorrecting chart (referred to as a grid chart hereinafter) in which gridpoints (dots) are arranged in a grid pattern. FIG. 2B schematicallyillustrates tile images formed on the imaging plane when the grid chartis imaged by the image pickup system according to this embodiment.

As illustrated in FIG. 2B, the tile images corresponding to the lensunits ML11, ML12, ML21, and ML22 are formed in sections T11, T12, T21,and T22 shown by the broken lines on the imaging plane, respectively.For better understanding, a distortion amount is exaggerated, and thegrid points in the grid chart image are connected by lines. The centerposition of the tile image is expressed by a symbol X. The distortioncorrection using a grid chart is realized by searching for a set of acorrection function and a coefficient in which the positions of the gridpoints in the grid chart image are orthogonally returned. In order tocorrect the distortion of the tile image, this embodiment divides thetile image from the entire image acquired by the image sensor SS, andgenerates a set of the correction function and the coefficient in thecoordinate system based on the center position of the tile image. Thenumber of sets of the correction function and the coefficient is as manyas the number of tile images. The information (array) obtained by thecorrection function and the coefficient indicates to which coordinatethe original coordinate in the tile image including the distortionshould be corrected. Information Y_(t) on the position of the correctedtile image in the tile coordinate system in the tile image is expressedby the following equation (1):Y _(t) =F _(t)(X _(t) −X _(ct))  (1)where t is the tile image number (tile number), X_(t) is information onthe position of the tile image in the sensor coordinate system (firstcoordinate system) in the imaging plane, X_(ct) is information on thecenter position of the tile image in the sensor coordinate system, andF_(t) is the correction function (including the coefficient) forcorrecting the tile image.

Given the information X_(ct) on the center position of the tile imageand the correction function F_(t) as a result of the preliminarycalibration, the information Y_(t) on the position of the corrected tileimage can be obtained. In the subsequent imaging, the distortion iscorrected for each tile image based on the information Y_(t) on theposition of the corrected tile image, and the multiband image isgenerated by superimposing images based on the corrected tile images.Since the number of divisions of the tile image and the number ofdistortion correcting calculations increase in proportion to the numberof tile images, the multiband image generating speed decreases. When amotion image is generated using the multiband image and the generatingspeed of a single multiband image (1 frame) decreases, the frame ratedecreases.

A description will now be given of the distortion correcting method thatrestrains the multiband image generating speed from lowering even if thenumber of tile images increases. FIGS. 3A to 3C schematically illustratea data structure of the multiband image. When images based ontwo-dimensional tile images are stacked in order of wavelength, themultiband image can be expressed by a three-dimensional data structureusing an x-axis and a y-axis for representing the spectroscopic imageand a λ-axis for representing the wavelength, as illustrated in FIG. 3A.This three-dimensional data structure will be referred to as a “datacube.” FIG. 3A illustrates an ideal data cube. In the data cubeillustrated in FIG. 3A, the distortion and the like are sufficientlycorrected, and the positional information is correct (or is notshifted). Each pixel value (luminance) in the multiband image discretelyexists on the three-dimensional coordinate grid point. This is similarlyapplied to any of the x-axis, y-axis, and λ-axis, and it can beinterpreted that only the number of divisions (resolution) is different.

The expression (1) is an operation formula that converts the informationX_(t) on the position of the tile image in the sensor coordinate systeminto the information Y_(t) on the position of the corrected tile imagein the tile coordinate system. In the expression (1), the correctionfunction F_(t) corresponds to a correction coefficient, and theinformation X_(ct) on the position of the center position of the tileimage in the sensor coordinate system corresponds to an offset. FIGS. 3Band 3C illustrate pre-correction and post-correction tile images,respectively. A multiband image is generated by stacking images based onthe corrected tile image in the wavelength order. Internal processingcorresponds to mapping all pixels on the image based on the correctedtile image onto the three-dimensional image. This mapping can berealized given the correspondence information between the tile number tin the expression (1) and the coordinate (layer position) on the λ-axis.In summary, the destination (address on the data cube) of each pixelvalue on the imaging plane is determined once the correction function isdetermined. Information Z on the position in the data cube coordinatesystem is expressed by the following expression (2):Z=GF(X−X _(c))  (2)where X is information on the position of the tile image in the sensorcoordinate system, X_(c) is information on the center position of thetile image in the sensor coordinate system, F is a matrix representingthe distortion correction (coordinate transformation) of the tile image,and G is a matrix for converting the tile coordinate system into thedata cube coordinate system (second coordinate system) in the multibandimage.

The expression (2) can be used to convert the information X on theposition of the tile image in the sensor coordinate system into theinformation Z on the position in the data cube coordinate system withoutusing the tile number t. Since the matrices F and G are lineartransformation matrices, the expression (2) can be turned into thefollowing expression (3):Z=H(X−X _(c))  (3)where H (conversion information) is a matrix that collectively expressesthe matrices F and G for converting one sensor coordinate system intothe data cube coordinate system.

By using the expression (3), the image acquired by a single calculationcan be divided into tile images and expanded on the data cube. Sinceonly the calculation for the sensor coordinate system is required, acalculation amount is constant without being affected by the increase ordecrease of the number of tile images. By acquiring the matrix H and theinformation X_(c) generated during the calibration or the information Xand Z, the acquired image can be converted into a multiband image at ahigh speed. Since the above calculation is a linear transformation, itcan be easily incorporated into a parallel computer (GPU or FPGA).

FIG. 4 is a flowchart showing a method of converting an image acquiredin this embodiment into a multiband image.

In the step S101, the processor 100 (acquisition unit 100 a) acquires animage made by capturing a grid chart image from the image sensor SS. Inthis embodiment, the grid chart illustrated in FIG. 2A is used.

In the step S102, the processor 100 (acquisition unit 100 a) acquiresinformation on the positions of the grid points of the grid chart in thesensor coordinate system. The information on the positions of the gridpoints is distortion information. In this embodiment, a dot is disposedat the grid point, but its type is not limited as long as distortioninformation can be obtained. A dot coordinate can be obtained at arelatively high speed using a Hough transform or the like.

In the step S103, the processor 100 (acquisition unit 100 a) acquiresinformation on the center position of the tile image in the sensorcoordinate system. The processor 100 may calculate the information onthe center position of the tile image using the information on thepositions of the grid points for each division, or may acquire theinformation from another means. The grid chart may be captured so as tofill the angle of view of each tile image, and the center coordinate ofthe grid chart may be substituted for information on the center positionof the tile image. This embodiment uses a grid chart consisting of 9×9grid points, and thus the center coordinate of the grid point.

In the step S104, the processor 100 (acquisition unit 100 a) acquiresstack information which is information on the correspondence between theinformation on the center position of the tile image in the sensorcoordinate system and the spectral wavelength information (layerposition in the spectroscopic image in the multiband image).

In the step S105, the processor 100 (acquisition unit 100 a) acquires acorrection function (including a coefficient) for correcting the tileimage.

In the step S106, the processor 100 (acquisition unit 100 a) acquiresthe matrix H as the conversion information used to convert into the datacube coordinate system the sensor coordinate system generated based onthe information on the center position of the tile image in the sensorcoordinate system, the stack information, and the correction function.The processing unit 100 b may generate the matrix H.

In the step S107, the processor 100 (processing unit 100 b) generates amultiband image using the object image acquired from the image sensorSS, and the matrix H.

In this embodiment, the distortion information of the optical system isacquired by capturing the grid chart with an actual machine, but may beacquired by using a simulation or a design value. If the manufacturingerror of the optical system can be sufficiently reduced, the distortionmap calculated for each tile image may be used as information on thepositions of the grid points for each tile image. As long as theinformation on the center position of the tile image in the sensorcoordinate system can be accurately acquired, the information generatedfrom the optical design value of the lens unit may be used as theinformation on the positions of the grid points in the tile coordinatesystem.

As described above, the configuration of this embodiment can acquire agood combined image (multiband image) at a high speed.

Second Embodiment

A basic configuration of the image pickup system according to thisembodiment is the same as that of the first embodiment. This embodimentwill discuss only the differences from the first embodiment.

The configuration according to this embodiment simultaneously corrects aluminance decrease around the object image or so-called shading. Thisembodiment corrects shading based on not only vignetting of the opticalsystem and the sensitivity characteristic of the image sensor SSrelative to the light incident angle, but also the transmissioncharacteristic of the bandpass filter relative to the incident angle.

The simplest shading correction is a method for capturing a white objecton the entire surface and for generating a luminance correcting map(correction information) backwardly calculated so that the in-planeluminance distribution of each tile image becomes constant. Thisembodiment uses the filter array FA including different types ofbandpass filters, and thus the white color here may be white(reflectance is constant within the use wavelength) over the entire usewavelength. The luminance correcting map may not be divided for eachtile image and may be used in the form of the sensor coordinate system.When the pixel value is coordinate-converted using the expression (3),the luminance correcting map is simultaneously coordinate-converted andthereby shading is simultaneously corrected. Before the distortioncorrection (coordinate conversion) is performed, the shading may becorrected on the entire image previously acquired from the image sensorSS by using the luminance correcting map. This method only performs asimple multiplication, has a small calculation load, and is not affectedby an increase or decrease of the number of tile images.

As described above, the configuration according to this embodiment cancorrect shading at a high speed, in addition to the effect of the firstembodiment.

Third Embodiment

The first embodiment has discussed a method of correcting the chromaticaberration generated in the lens array, but this embodiment will discussa method of correcting different aberrations.

FIG. 5 schematically illustrates an image pickup system according tothis embodiment. The image pickup system includes an optical system, animage sensor SS, and a processor 100. The optical system includes a lensL1, a filter array FA (omitted in FIG. 5 for simple explanation), and alens array LA.

When a common optical system such as the lens L1 (referred to as a “mainoptical system” hereinafter) is disposed on the object side of the lensarray LA, optical axes (labelled as AX1 and AX2 in FIG. 5 ) of the lensunits in the lens array LA shift from an optical axis (main opticalaxis) AX0 of the main optical system. Thus, many lens units in the lensarray LA have different eccentric distortions depending on the distanceand direction from the main optical axis AX0. In an attempt to correctan eccentric distortion that differs for each tile image, a calculationload will increase because it is necessary to calculate a displacementin the information on all the positions in the tile image, and a largememory capacity is necessary to store the information on the distortedposition.

This embodiment reduces a data amount from the viewpoint of geometricaloptics. A shift amount (eccentric component) of the lens unit in thevertical direction from the main optical axis AX0 is considered as avector and is defined as an “eccentric amount ε” When the eccentricity εis 0, that is, when the optical axis of the lens unit is located on themain optical axis AX0, the distortion of the tile image captured by thelens unit becomes symmetrical with respect to the optical axis, so thecorrection function can be described with a single variable as an angleof view ω. The distortion of the tile image formed by the lens unit whenthe eccentricity ε is 0 will be referred to as a “reference distortion.”

It is known that the eccentric distortion is proportional to the productof the eccentric amount ε and the square of the angle of view ω. FIG. 6explains the eccentric distortion. In FIG. 6 , the displacement amountis exaggerated. The image is deformed (eccentrically distorted) by theamount and direction determined by the eccentric amount ε. The magnitudeof the deformation amount can be expressed by a coefficient. In summary,the total distortion of the tile image formed by the lens unit disposedat the eccentric position can be approximated by the product of thereference distortion component and the eccentric distortion component. Atile image correction function f_(di) is expressed by the followingexpression (4):F _(di)(x _(i) ,y _(i))=f ₁(ω_(i))×f ₂(ε_(i))  (4)where i is a tile number, x_(i) and y_(i) represent information on thecorrected position in the tile image in the sensor coordinate system,ω_(i) is an angle of view in the tile image, and ε_(i) is an eccentricamount of the lens unit from the main optical axis corresponding to thetile image, f₁ is a function including a reference distortion component(angle of view component), and f₂ is a function including an eccentricdistortion component (shift amount component).

Backwardly calculating and assigning the coefficients suitable for thefunctions f1 and f2 can generate a correction function for the eccentricdistortion of the tile image. Thus, the correction amount of thereference distortion is previously calculated and stored asone-dimensional data of the angle of view ω, and thereby the distortioncorrection amount of the tile image can be generated by a simplecalculation.

As described above, the configuration according to this embodiment cancorrect an eccentric distortion with a small data amount.

Fourth Embodiment

As described in the first embodiment, the lateral chromatic aberrationgenerated in the lens unit in the lens array LA can be collectivelytreated as the distortion in the image processing, and thus can becorrected by the same function as that of the distortion correction. Thelateral chromatic aberration is generally expressed by the differencefrom the image height of the reference wavelength. The correctionfunction of the tile image is expressed with a function f_(ci)representing a distortion component of the tile image at a referencewavelength where an arbitrary tile image is imaged at the referencewavelength, and a function f₃ representing a distortion component(deformed component) caused by the lateral chromatic aberration of thespectral wavelength. The distortion correction function f_(di) of thetile image is expressed by the following expression (5):f _(di)(x _(i) ,y _(i))=f _(ci)(x _(0i) ,y _(0i),λ₀)×f ₃(λ_(i))  (5)where i is a tile number, x_(i) and y_(i) include information onpost-correction position in the tile image in the sensor coordinatesystem, x_(0i) and y_(0i) include information on pre-correction positionin the tile image in the sensor coordinate system, λ₀ is the referencewavelength, and λ_(i) is an imaging wavelength of the tile image.

Since the distortion component caused by the lateral chromaticaberration can be treated independently of the eccentric distortion inthe third embodiment, the distortion correction function f_(i)(x_(i),y_(i)) is expressed by the following expression (6) in combination withthe third embodiment.f _(di)(x _(i) ,y _(i))=f ₁(ω_(i))×f ₂(ε_(i))×f ₃(λ_(i))  (6)where f_(di) is a distortion correction function of the tile image (i),x_(i) and y_(i) include information on post-correction position in thetile image (i) in the sensor coordinate system, f₁ is a functionrepresenting a reference distortion component, ω_(i) is an angle of viewin the tile image, f₂ is a function representing an eccentric distortioncomponent caused by an eccentric amount ε_(i), and ε_(i) is an eccentricamount (shift amount) [mm] from the main optical axis in the verticaldirection.

As described above, the configuration according to this embodiment cancorrect the lateral chromatic aberration with a small data amount.

Each embodiment corrects the distortion generated by the filter arrayFA, but the present invention is applicable to a correction of thedistortion generated by the lens array LA. For example, when the lensunits in the lens array LA are manufactured with different designs, thedistortions generated in the plurality of tile images are different fromeach other.

The above embodiment can provide a processing apparatus, an image pickupapparatus, an image pickup system, and a processing method, each ofwhich can generate a good combined image.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2020-097907, filed on Jun. 4, 2020 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A processing apparatus configured to generate acombined image by stacking a plurality of images based on a plurality ofobject images formed on an imaging plane of an image sensor by aplurality of lens units, the processing apparatus comprising: at leastone processor or circuit configured to execute a plurality of tasksincluding: an acquisition task configured to acquire conversioninformation for converting a first coordinate system in the imagingplane into a second coordinate system in the combined image, theconversion information being generated based on information on a centerposition of each of the plurality of object images on the imaging plane,information on a correspondence relationship between the centerpositions and layer positions of the plurality of images in the combinedimage, and a correction function for correcting the plurality of objectimages; and a processing task configured to generate the combined imageusing the conversion information.
 2. The processing apparatus accordingto claim 1, wherein the acquisition task acquires correction informationfor correcting an in-plane luminance distribution of an imagecorresponding to each of the plurality of object images, and wherein theprocessing task corrects an image corresponding to each of the pluralityof object images using the correction information.
 3. The processingapparatus according to claim 1, further comprising a memory configuredto store the conversion information.
 4. An image pickup apparatuscomprising: the processing apparatus according to claim 1; and an imagesensor configured to generate the plurality of images.
 5. The imagepickup system comprising: the image pickup apparatus according to claim4; and an optical system that includes a plurality of lens units.
 6. Theimage pickup system according to claim 5, wherein the optical systemincludes a plurality of filters disposed on optical axes of theplurality of lens units and having transmission characteristicsdifferent from each other.
 7. The image pickup system according to claim6, wherein the correction function includes a function including adeformation component caused by a lateral chromatic aberration of eachof the plurality of object images, and a function including a distortioncomponent of each of the plurality of object images at a referencewavelength.
 8. The image pickup system according to claim 5, wherein theoptical system includes a main optical system on an object side of theplurality of lens units.
 9. The image pickup system according to claim8, wherein the correction function includes a function including anangle of view component of each of the plurality of object images, and afunction including a component of a shift amount between an optical axisof the main optical system and an optical axis of each of the pluralityof lens units.
 10. A processing method for a processing apparatusconfigured to generate a combined image by stacking a plurality ofimages based on a plurality of object images formed on an imaging planeof an image sensor by a plurality of lens units, the processing methodcomprising the steps of: acquiring conversion information for convertinga first coordinate system in the imaging plane into a second coordinatesystem in the combined image, the conversion information being generatedbased on information on a center position of each of the plurality ofobject images on the imaging plane, information on a correspondencerelationship between the center positions and layer positions of theplurality of images in the combined image, and a correction function forcorrecting the plurality of object images; and generating the combinedimage using the conversion information.